When applications were first developed for the graphical user interface (GUI) (e.g., Apple Macintosh, Microsoft Windows, UNIX Motif and IBM OS/2 Presentation Manager). application developers used standard GUI objects as building blocks to construct the user interface of the software applications being developed. The standard GUI objects are used to display information to the user and are provided by the GUI vendors. The GUI vendors provide a published interface, including a class, behaviors and properties which allow other programs to send events to objects in order to navigate through the application and to get information from objects to examine application behavior. Examples of standard objects include Checkbox, Listbox, Textfield and Treeview. Essentially, there are a small number of standard controls with a limited number of properties and methods.
GUI applications include some body of executable code which presents data to a user and accepts user responses in the form of keyboard entries or mouse events. Graphical primitives are provided by the GUI operating system and include functions that draw text, lines and images. When the graphical primitives are called, they cause the visual representation of the data that the user actually views. Graphical primitives are called with arguments that specify the desired characteristics of the graphical request. For example, to request that the graphical operating system draw a line, a call is made to the DrawLine primitive function as follows: "DrawLine (hWnd=0B38, Horizontal at (top=341, left=70), size=101). This call results in the drawing of a horizontal line 101 pixels in length starting at the x-y coordinate 341, 70. Similarly, to draw text, a call is made to the graphical primitive DrawText as follows: "DrawText (hWnd=0B38, Rect=(top=320, left=84, bottom=339, right=158), `Last Trade`). This call results in the display of the string "Last Trade" from the x-y coordinate 320, 84 to the x-y coordinate 339, 158. More complex graphical objects such as listbox, combobox, treeview and table are drawn using these graphical primitive functions.
As more and more applications have been developed for the GUI, and more recently for the World Wide Web, there has been a general trend away from standard objects to custom objects which allow the developer greater control and flexibility, as well as the opportunity to bring richness and novelty to the user interface. The trend away from standard objects is a move from a relatively small number of well defined GUI components to a much larger number of less well defined components. However, there still exists a need to characterize these custom objects so that application software can utilize them. Existing characterization schemes are not well suited to such a task. For example, optical character recognition (OCR) is generally capable of only recognizing, individual character images by matching a pattern of lines and curves to a small set of known characters. Also, screen-scraper technology is limited to recognizing text in a fixed number of rows and columns where color is the only attribute. Neither of these approaches is able to recognize and characterize a large number of graphical objects.